Skip to content

Add the io_uring link (E3LinkLayer::URING) - #167

Draft
Thecave3 wants to merge 7 commits into
feat/104-107-posix-one-endpointfrom
feat/140-uring-connector
Draft

Thecave3 wants to merge 7 commits into
feat/104-107-posix-one-endpointfrom
feat/140-uring-connector

Conversation

@Thecave3

@Thecave3 Thecave3 commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • A third link, E3LinkLayer::URING (--link uring): the POSIX link's endpoint, sockets and bytes (IPC, TCP and SCTP, as framed since posix: one endpoint per dApp, one PDU per message, and each PDU to its dApp's own connection #153), driven through Linux io_uring. A RAN of one link serves a dApp of the other; tested in both directions on all three transports, and against a real gNB (below).
  • The facade gains one enumerator (URING = 2, string uring, C API range 0..2, Python E3LinkLayer_URING). No new E3Config field or E3Connector method: PosixOptions apply as they are. Ring options are deployment tuning read from LIBE3_URING_* when the connector is created, not ABI.
  • Built when it can be (LIBE3_ENABLE_URING, default ON = Linux with liburing found; LIBE3_HAS_URING 0/1). Without it, or where the kernel refuses io_uring (Docker's default seccomp, kernel.io_uring_disabled, a kernel before 5.11), create_connector returns null with the reason, like ZMQ without libzmq. No silent fallback.
  • The calling threads drive two rings (receive, send); no I/O thread, so send() still returns when the kernel has the bytes and SEND_DONE compares row by row with POSIX. A broadcast is one batch: a stalled dApp no longer delays the others.
  • Opt-in busy polling (LIBE3_URING_BUSY_POLL_US) and kernel submission thread (LIBE3_URING_SQPOLL); registered files on, registered buffers off by default, each with a fallback.
  • One behaviour change in the POSIX link, in a commit of its own (connector: bound the retransmission timeout of every SCTP socket): each SCTP association bounds its retransmission timeout (120 ms first, 20 to 250 ms after) instead of Linux's 1 s minimum. A burst of 2000 PDUs of up to 64 KiB dropped the dApp (a send stalled past the 500 ms timeout) in 8 of 100 runs on POSIX and 7 of 100 on io_uring; 0 of 100 on both with the bound. test_socket_link reads the values back. Drop that commit if the POSIX timers should stay as posix: one endpoint per dApp, one PDU per message, and each PDU to its dApp's own connection #153 left them.
  • The socket setup and framing both links share moved from posix_connector.cpp to socket_link.{hpp,cpp} with no change of behaviour (the POSIX tests are unchanged and pass).
  • Test infrastructure fixed on the way, each with its cause: the tests' ports now come from free_ports(), which reserves them with a lock file (a port from the range the kernel uses for source ports collided with a TIME_WAIT one, 2 runs in 30 of test_uring_connector; and three test files derived their ports from the pid with bands of their own that overlapped, so test_role_pair_posix failed in 8 runs of 12 beside the others; 0 in 80 and 0 in 12 after); the link-matrix CTest entries run with LATREC_ENTRIES_LOG2=12 (the 8 MiB pre-faulted latrec ring of every agent thread filled a disk when the suite was repeated); the full-loop benchmark's pairing check allowed no skew between a send stamp and the receive stamp of a receiver that polls, and refused about one run in ten of SCTP with busy polling although the counts of PDUs out and in agreed (it now allows 5 us, says which pair was first, and its arithmetic is signed).
  • Architecture, decisions and measurements: docs/connectors/uring.md.

Which link is fastest

tools/bench_matrix.py, nine interleaved rounds of 3000 round trips per cell, ASN.1, a Linux 6.8 virtual machine of 12 vCPUs on an Intel Xeon Gold 6240R server (nothing pinned or isolated; the cluster's host kernels disable io_uring, so it had to be a VM); round trip p50 in µs, CPU per round trip in brackets:

Transport zmq posix uring uring + busy polling uring + SQPOLL + busy polling
IPC 88.8 (186) 50.4 (138) 44.1 (128) 26.2 (149) 20.7 (194)
TCP 98.9 (204) 47.3 (129) 49.6 (132) 34.5 (183) 27.3 (235)
SCTP - 48.6 (139) 54.2 (148) 43.0 (217) 33.1 (271)
  • Fastest: io_uring with the kernel submission thread and busy polling, on every transport and encoding (JSON and Protobuf too): 59, 42 and 32% below POSIX with ASN.1, at 1.4 to 2.0 times its CPU and a p99 27 to 51% below POSIX's. It needs cores of its own. On the x86 server the gain is the connector's own rows (the wire hop 7.0 to 1.6 µs, the send 4.3 to 2.5 µs); on an Arm virtual machine (earlier campaign, in the doc) the same variant was first too (19 µs against 110) but mostly because no thread in the loop slept any more, and its p99 was worse than POSIX's.
  • Busy polling alone is 48, 27 and 12% below POSIX on IPC, TCP and SCTP.
  • Default io_uring is POSIX within the noise (12% below on IPC, 5 and 12% above on TCP and SCTP, the rounds overlap). Registered files and buffers: no measurable effect. LIBE3_URING_SQPOLL without busy polling is not a general choice (29% below POSIX on IPC, the same on TCP, 16% above on SCTP).
  • Full tables (both machines, JSON and Protobuf, the 18-phase breakdown) in docs/connectors/uring.md.

Type of change

  • Bug fix
  • New feature / enhancement
  • New Service Model
  • Refactor (no behavior change)
  • Documentation
  • Test / CI / packaging
  • Other (explain):

Linked issue

Closes #140.

Decisions taken without the owner (the issue put three to him)

Issue decision Taken What it makes appear What the other option would have made appear
1. Where the ring is driven (a) by the calling thread, one ring per direction send() returns once the kernel has the bytes, as POSIX; latrec rows compare one to one (b) an I/O thread: send() returns on enqueue, a thread hop on every PDU (a wakeup, 15 to 20 µs on the Arm test machine)
2. Wire framing the POSIX wire, which #153 already length-prefixes on TCP in both directions mixed RAN/dApp pairs work; no ledger row for the wire (nothing a peer can observe changes) a uring-only wire: two camps of peers
3. Default of LIBE3_ENABLE_URING auto: ON, built when Linux and liburing allow a status line at configure, never a failure hard failure without liburing

Also: per-thread rings with SINGLE_ISSUER/DEFER_TASKRUN not done (the facade lets several threads read and send). Consequence written in the doc: a socket call with a timeout on the RAN's inbound thread (where SM control handlers run) can return EINTR while other dApps' receives are armed. docs/interop.md has a ledger row for the new link and for the SCTP timer bound. No VERSION bump (0.6.0 is unreleased).

Testing

Linux 6.8 on x86-64 (a KubeVirt VM), on the final sources:

  • Whole suite with integration tests, SCTP and latrec, LIBE3_REQUIRE_URING=1 LIBE3_REQUIRE_SCTP=1: 81/81. ./build_libe3 -c -r --all-encodings --enable-sctp -t from scratch: 62/62. Each of the first six commits was built on its own; the connector tests passed on the first four. The six were then run again as one tree (the seventh changes documents only): 81/81 as above, and the 28 port-using tests ten times over at -j2 and at -j6 (--repeat until-fail:10): 0 failures in each.
  • test_uring_connector alone 80 times, no failure (2 in 30 before the port fix). The connector subset (21 tests at -j3) 12 times, no failure; before the second port fix it failed in 8 of 12, always test_role_pair_posix or a variant (see below). The same on an Arm VM: the io_uring and link-matrix tests 15 times each, clean.
  • ASan + UBSan + LeakSan on the connector tests: 25/25, 0 reports. TSan: 20 reports, all inside libzmq's own threads, none in io_uring, socket_link or POSIX code; the two tests that fail under it are the ZeroMQ ones, as on Arm.
  • example_simple_agent + example_simple_dapp as two processes: ZMQ, POSIX and io_uring on each transport and both mixed directions on IPC, TCP and SCTP: 14/14.
  • Extended, with a real gNB: the OAI gNB with its E3 agent (--E3Configuration.link uring) and one nrUE on the RF simulator, and the dApp library's spectrum dApp, on IPC, TCP and SCTP with the gNB and the dApp each on POSIX or io_uring: 12/12 pairs run, 16,000 to 18,600 sensing slots handled in 15 s, 99.7 to 99.9% of them with their sensing ranges, no assertion in the gNB. The dApp library's own tests: 141 passed, 4 skipped, with libe3py built from this branch. Two small patches for the twins (below) came out of it.
  • Strict Doxygen as CI's docs job: 0 warnings in changed files (checked on Arm; unchanged since except the docs). The E3AP Word edition regenerated (one added paragraph; TOC page numbers shift). SWIG smoke test passes with E3LinkLayer_URING == 2.
  • TDD: the new tests did not compile before URING existed; the stalled-peer test caught the EINTR hazard of decision 6; test_socket_link fails with the SCTP timer block removed; test_bench_matrix fails without the signed parser.

One test failure I could not explain, not in this PR's code: in a stress run (ctest -j2 --repeat until-fail:15 over 29 tests) test_latrec_stages failed once in its posix-ipc-asn1 combination: the RAN's outbound queue (capacity 128) was full for about a second and 200 indications were dropped (Buffer too small), with nothing logged by the connector. It did not reproduce in 100 runs of the same test at this branch and 100 at #153's head under the same load. The POSIX path and the queue are what #153 left; I have not touched them beyond the shared-socket move. Worth a look if it shows up in CI.

Not run: GitHub's runners (the CI changes are untested until a PR runs them; the first run rebuilds the asn1c cache because build_libe3 changed); bare metal with isolated cores; two hosts; OCUDU's agent; a gNB with a core network.

Found on the way, not changed here

  • An IPC endpoint is created with the permissions of the RAN's user (srwxr-xr-x root root for a gNB under sudo), so a dApp of another user cannot connect to it. The same on the POSIX link. A one-line chmod in socket_link::listen_on would change it; it is a decision about who may talk to a RAN, so I left it.
  • The dApp library's spectrum dApp subscribes the L1-KPM function with control identifier 1, which that function does not have; libe3's refusals ([Interop]: A libe3 RAN agent accepts subscriptions to RAN functions and telemetry ids that do not exist #96) now answer unsupportedIdentifier. A dApp-library fix.

Twin-repo coordination

  • This PR does not change the E3 wire protocol or public ABI, OR a paired PR exists in each affected twin repo (link below).
  • This PR does not change the wire format or a behavior a peer (OAI, Aerial, OCUDU) can observe, OR I added a row to the change ledger in docs/interop.md.

The ABI change is additive (one enumerator, a wider C API range). A twin only needs a change to offer uring in its own configuration parsing; nothing breaks without it. Both changes are on branches, built and run as above: the dApp library (E3LinkLayer.URING and a link map that offers uring only when the binding has it) and OAI (uring among the E3 configuration's accepted strings and the README). No pull request is open for either.

Base

Stacked on #153 (feat/104-107-posix-one-endpoint); seven commits (the last one is the section on switching io_uring on for one group, in the README and docs/connectors/uring.md).

… posix_connector

The endpoint parsing, the creation and tuning of a socket, listen, connect and admit,
the TCP length prefix, the SCTP send info and the test for an SCTP notification now live
in socket_link.{hpp,cpp}, so a second link can put the same bytes on the same sockets.
posix_connector.cpp keeps what is its own: the threads, the queues, the per-peer state.

Nothing a peer can observe changes and no POSIX test changed. test_socket_link pins the
framing the two links must share: a whole PDU, a partial one, one of the limit and one a
byte over it, the prefix written, and the Invalid-PDU-length message the POSIX link logs.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Andrea Lacava <a.lacava@northeastern.edu>
Linux gives an SCTP association a minimum retransmission timeout of one second. On
loopback a burst of large PDUs can lose one packet to a full receive window, the
association then waits that second, and the send stalls past the 500 ms send timeout: the
RAN drops a dApp that is alive. A burst of 2000 PDUs of up to 64 KiB from the RAN, repeated
100 times, dropped the dApp 8 times with the POSIX link and 7 times with the io_uring link of
the next commits, and never over TCP; with
SCTP_RTOINFO set on the socket (initial 120 ms, minimum 20 ms, maximum 250 ms, the maximum
kept under the send timeout) it happened 0 times in 100 with each link.

This changes the timers of #153's POSIX SCTP sockets too, for the better as far as the
runs show, but it is a choice about a transport's behaviour, which is why it is a commit
of its own: dropping it removes the bound and nothing else. test_socket_link reads the
values back from the socket and fails without it.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Andrea Lacava <a.lacava@northeastern.edu>
A third link beside ZeroMQ and POSIX: the POSIX link's endpoint, sockets and bytes (IPC,
TCP and SCTP, one endpoint per dApp, the framing of the one-endpoint wire), driven
through Linux io_uring. A RAN on one of the two socket links serves a dApp on the other.

For a user the facade gains one enumerator: URING = 2, the string "uring", C API range
0..2, E3LinkLayer_URING in Python. No E3Config field and no E3Connector method is added
and PosixOptions apply as they are. The ring is tuned by LIBE3_URING_* environment
variables read when the connector is created (depth, registered files, registered buffers,
a kernel submission thread, busy polling of the completion queue): deployment tuning, not
ABI.

The calling threads drive two rings, one for receives and one for sends, with no I/O thread
of the connector's own, so send() returns when the kernel has the bytes, as it does with
POSIX, and the SEND_DONE stamp compares row by row. A broadcast to several dApps is one
batch of sends, so a dApp that stopped reading no longer delays the others for the send
timeout. uring_ring.{hpp,cpp} is the only file that includes <liburing.h>.

The link is built when it can be: LIBE3_ENABLE_URING (default ON) needs Linux and liburing,
and LIBE3_HAS_URING is 0 or 1 for what links libe3. Without liburing, or where the kernel
refuses io_uring (Docker's default seccomp profile, kernel.io_uring_disabled, a kernel
before 5.11), create_connector returns null and says why, as it does for ZeroMQ without
libzmq: there is no silent fallback. build_libe3 learns --enable-uring and --disable-uring
and installs liburing-dev with its other dependencies.

The decisions, what was left out and why (per-thread rings with SINGLE_ISSUER and
DEFER_TASKRUN, a thread of its own for I/O, a direct-send bypass, multishot receive) are in
docs/connectors/uring.md.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Andrea Lacava <a.lacava@northeastern.edu>
…_uring link, compare the links

The tests of the one-endpoint wire take --ran-link and --dapp-link (tests/link_under_test.hpp)
and run once per pair in CTest: both POSIX, both io_uring and each mixed pair. Where the
kernel refuses io_uring they exit 77, which CTest reports as skipped, unless
LIBE3_REQUIRE_URING is set, so that a machine meant to run io_uring cannot pass by skipping.
test_uring_connector covers the ring options, a broadcast to more peers than the ring has
entries, a stalled peer, a reset peer, 2000 PDUs of every size on every transport, the PDU
limit, the receive timeout and the wake of a waiting receive by shutdown.

The ports of these tests now come from free_ports(): below the range of source ports, free when
asked, and claimed with a lock on a file in /tmp/libe3-test-ports that the process holds until it
exits, so that two tests never take the same port however they order asking and listening. The
tests chose their ports from the range the kernel also uses for the source ports of outgoing
connections, and each from a formula on its pid with a base and a width of its own, so the bands
overlapped. Once enough connections had come and gone the listen of a later test met a source port
still in TIME_WAIT (2 runs in 30 of test_uring_connector), and two tests with overlapping bands met
each other (8 runs in 12 of the connector subset at -j3, always test_role_pair_posix or a variant).
0 in 80 and 0 in 12 after.

The link-matrix entries run with LATREC_ENTRIES_LOG2=12: every agent thread leaves an
8 MiB pre-faulted latrec ring in <build>/latrec, and repeating the matrix filled a disk.

test_bench_full_loop_latency takes --link uring, --iterations, --warmup and CPU pinning,
and reports the CPU and context switches per round trip. tools/bench_matrix.py runs it over
links, transports and encodings in interleaved rounds and names the fastest link per
transport only when its slowest round beats every other link's best; its parsing and verdict
are tested on canned output (test_bench_matrix.py). The benchmark pairs the dApp's inbound
records with the RAN's outbound ones by position and checked that each pair was in order;
a receiver that polls can have a PDU before the sender has stamped its send as done (SCTP
over loopback delivers inside the send call), so about one run in ten of SCTP with busy
polling was refused as misaligned although the counts of PDUs out and in agreed. The check
now allows 5 us of skew (a pair that is off by one round trip is off by 12 us or more), the
phase arithmetic is signed, and a refusal says which pair was first and how many groups
there were.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Andrea Lacava <a.lacava@northeastern.edu>
…he others

The E2E dApp job runs the links [zmq, posix, uring] over [ipc, tcp, sctp] (ZeroMQ has no
SCTP) with SCTP built in; the unit and integration jobs set LIBE3_REQUIRE_URING=1 and print
the kernel's io_uring settings first, so a runner that refuses io_uring fails instead of
skipping; one job builds without liburing and checks that the link is reported as not built.
The full-loop job runs tools/bench_matrix.py over ZeroMQ, POSIX, io_uring, and io_uring with
busy polling with and without the kernel submission thread, on IPC, TCP and SCTP, three
rounds, and appends the table to the CI report.

The images carry liburing2, and run-pair.sh starts a container with seccomp unconfined when
the link is uring (Docker's default profile refuses io_uring_setup). example_simple_agent and
example_simple_dapp accept --link uring.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Andrea Lacava <a.lacava@northeastern.edu>
docs/connectors/uring.md says what a user can do with the link, which link to pick, the
architecture and the eleven decisions with what the others would have made of it, where it
runs, the ring options, notes for a real-time deployment, the tests, the measurements on an
x86-64 server and on an Arm virtual machine (every number is in the page and in the tools'
CSV output, nothing is rounded to a story) and what was deferred with its cost.

The README, CLAUDE.md, the path-A control-loop page (the wire hop can read below zero), the
integrator guide (the link and its framing code pointers, the SCTP timer bound), the
encoding chapter of the specification (one sentence: how a link moves the bytes is not part
of E3AP; the Word edition is rebuilt), interop.md (rows of the pairs table, a mixed-links
note, a ledger row for the new link and for the SCTP timer bound), and the SWIG README are
updated.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Andrea Lacava <a.lacava@northeastern.edu>
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

CI report — d429af7 — ✅ all checks passed

Workflow Result Time Run
CI ✅ success 32m55s #20
Docs ✅ success 1m03s #94
CI

✅ posix/ipc

  • dApp exit: 0
  • Indications received: 7

✅ posix/sctp

  • dApp exit: 0
  • Indications received: 7

✅ posix/tcp

  • dApp exit: 0
  • Indications received: 7

✅ uring/ipc

  • dApp exit: 0
  • Indications received: 7

✅ uring/sctp

  • dApp exit: 0
  • Indications received: 7

✅ uring/tcp

  • dApp exit: 0
  • Indications received: 7

✅ zmq/ipc

  • dApp exit: 0
  • Indications received: 7

✅ zmq/tcp

  • dApp exit: 0
  • Indications received: 7

zmq/ipc

  • ✅ 1 RAN - 1 dApp: indications=5
    • dapp peer=t11 ran=ran-solo sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
  • ✅ 1 RAN - 2 dApps: dApp#1 ind=5 sub=1, dApp#2 ind=6 sub=2, RAN saw 2 dApps
    • dapp1 peer=t12 ran=ran-shared sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
    • dapp2 peer=t12 ran=ran-shared sub=2 indications=6 seq=[0..5] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:6 2-5:0 6-10:0 >10:0]
  • ✅ 2 RANs - 1 dApp: from ran-a ind=5, from ran-b ind=5
    • dapp peer=t2a ran=ran-a sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
    • dapp peer=t2b ran=ran-b sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]

zmq/tcp

  • ✅ 1 RAN - 1 dApp: indications=5
    • dapp peer=default ran=ran-solo sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0.8 max=1 @seq=1) hist[<=1:5 2-5:0 6-10:0 >10:0]
  • ✅ 1 RAN - 2 dApps: dApp#1 ind=5 sub=1, dApp#2 ind=6 sub=2, RAN saw 2 dApps
    • dapp1 peer=default ran=ran-shared sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
    • dapp2 peer=default ran=ran-shared sub=2 indications=6 seq=[0..5] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:6 2-5:0 6-10:0 >10:0]
  • ✅ 2 RANs - 1 dApp: from ran-a ind=5, from ran-b ind=5
    • dapp peer=default ran=ran-a sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0.2 max=1 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
    • dapp peer=off100 ran=ran-b sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0.4 max=1 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]

Full-loop latency

Full-loop latency benchmark (N=1040 after 50 warmup)

All values in microseconds (μs). Link: zmq, transport: ipc, encoding: ASN.1 APER.

# Description Tags mean p50 p99 max
1 Collect indication data RECORD_BEGIN to ENCODE_E3SM_BEGIN 0.10 0.09 0.24 0.52
2 Create & encode indication ENCODE_E3SM_BEGIN to ENCODE_E3SM_DONE 0.86 0.83 1.43 9.94
3 Encode E3AP (indication) EMIT_ENTER to ENQUEUE, then DEQUEUE to ENCODE_E3AP_DONE 3.24 3.04 5.68 28.25
4 Queuing (indication) ENQUEUE to DEQUEUE 12.33 11.99 25.08 34.26
5 Delivery (indication) ENCODE_E3AP_DONE to SEND_DONE 6.55 7.14 10.17 24.80
6 E3 wire (RAN -> dApp) SEND_DONE to RECV 41.00 38.01 60.77 833.49
7 Decode E3AP (indication) RECV to DECODE_E3AP_DONE 2.69 2.45 15.41 26.94
8 libe3 dispatch (indication) DECODE_E3AP_DONE to DELIVER_BEGIN 0.16 0.14 0.24 16.62
9 Decode indication DELIVER_BEGIN to DECODE_E3SM_DONE 0.52 0.55 0.80 1.22
10 Process data DECODE_E3SM_DONE to ENCODE_E3SM_BEGIN 0.04 0.04 0.06 0.31
11 Create & encode control ENCODE_E3SM_BEGIN to ENCODE_E3SM_DONE 0.32 0.32 0.55 0.71
12 Encode E3AP (control) EMIT_ENTER to ENQUEUE, then DEQUEUE to ENCODE_E3AP_DONE 3.42 3.29 5.55 23.16
13 Queuing (control) ENQUEUE to DEQUEUE 11.96 11.10 25.31 36.16
14 Delivery (control) ENCODE_E3AP_DONE to SEND_DONE 6.06 4.84 22.29 36.22
15 E3 wire (dApp -> RAN) SEND_DONE to RECV 40.48 40.27 61.27 214.43
16 Decode E3AP (control) RECV to DECODE_E3AP_DONE 2.37 2.22 4.67 19.18
17 libe3 dispatch (control) DECODE_E3AP_DONE to DECODE_E3SM_BEGIN 0.59 0.58 1.02 1.26
18 Decode & handle control DECODE_E3SM_BEGIN to DECODE_E3SM_DONE 0.24 0.20 0.59 0.76
Total Total round-trip 133.35 127.31 179.87 947.89

CPU per round trip, both ends: 81.21 us user, 225.87 us system; context switches per round trip: 9.03 voluntary, 0.08 involuntary.

ubuntu-latest, Release build, ZMQ + IPC, ASN.1 APER.

Connector comparison: full-loop round trip

Same machine, 3 interleaved round(s) per cell, 1000 round trips per run after 50 of warm-up. Each value is the median over the rounds of that round's statistic, with the range of the rounds in brackets. Microseconds. CPU is user plus system time of the process (both ends) per round trip.

Transport Encoding Link Round trip p50 Round trip p99 Wire RAN to dApp p50 Wire dApp to RAN p50 Send indication p50 Send control p50 CPU per round trip Runs
ipc asn1 zmq 127.7 [125.6 to 128.4] 160.5 [156.7 to 162.4] 39.4 [38.9 to 39.6] 39.2 [39.0 to 39.7] 4.8 [4.8 to 5.0] 4.9 [4.8 to 4.9] 303.4 [301.9 to 303.8] 3
ipc asn1 posix 48.3 [44.8 to 49.1] 69.6 [68.1 to 72.5] 10.9 [10.3 to 11.3] 11.0 [9.7 to 11.3] 7.5 [6.7 to 7.6] 5.2 [4.1 to 7.1] 162.0 [155.4 to 163.3] 3
ipc asn1 uring 47.0 [39.5 to 47.2] 66.8 [60.5 to 68.6] 8.7 [7.7 to 9.2] 7.6 [7.5 to 8.5] 7.7 [5.8 to 9.0] 8.5 [5.7 to 9.4] 156.6 [146.1 to 158.2] 3
ipc asn1 uring-busy 29.3 [27.6 to 30.7] 42.9 [41.4 to 45.4] 3.0 [2.8 to 3.2] 3.2 [3.0 to 3.5] 5.2 [5.1 to 5.8] 5.0 [5.0 to 5.5] 148.2 [142.6 to 151.7] 3
ipc asn1 uring-sqpoll-busy 942.3 [892.3 to 1008.2] 1980.9 [1970.6 to 2988.1] 6.2 [6.2 to 7.4] 6.8 [6.6 to 7.4] 3.4 [3.2 to 3.5] 3.2 [3.1 to 3.3] 4230.4 [4191.1 to 4364.3] 3
tcp asn1 zmq 159.0 [158.4 to 159.4] 197.3 [196.8 to 197.9] 52.2 [51.9 to 53.6] 52.6 [52.0 to 53.0] 5.2 [5.1 to 5.7] 5.2 [5.1 to 5.4] 332.9 [332.9 to 333.3] 3
tcp asn1 posix 45.0 [44.6 to 64.0] 68.6 [67.1 to 88.6] 6.5 [6.4 to 6.9] 6.5 [6.4 to 8.3] 10.7 [10.7 to 19.5] 10.6 [10.5 to 13.9] 153.1 [152.9 to 185.3] 3
tcp asn1 uring 64.1 [60.4 to 64.6] 90.2 [87.7 to 91.4] 8.6 [7.5 to 8.8] 8.1 [7.0 to 8.9] 16.4 [14.5 to 17.6] 17.2 [12.0 to 19.3] 198.9 [197.1 to 202.5] 3
tcp asn1 uring-busy 39.8 [37.4 to 40.6] 56.5 [52.1 to 58.4] 1.5 [1.5 to 1.6] 1.8 [1.7 to 1.8] 12.1 [11.5 to 12.9] 11.0 [11.0 to 11.2] 192.1 [182.5 to 202.9] 3
tcp asn1 uring-sqpoll-busy 1027.2 [1007.7 to 1033.8] 1992.4 [1465.0 to 2002.5] 5.8 [5.6 to 5.9] 5.8 [5.6 to 6.0] 12.6 [9.2 to 13.3] 12.6 [9.3 to 13.1] 4245.5 [4244.0 to 4418.8] 3
sctp asn1 posix 69.1 [68.6 to 69.8] 93.6 [92.0 to 97.4] 3.1 [-0.6 to 3.2] -0.8 [-1.0 to 3.5] 21.9 [21.7 to 27.6] 29.6 [22.8 to 29.8] 213.1 [210.4 to 217.3] 3
sctp asn1 uring 72.7 [69.1 to 73.8] 97.9 [96.7 to 99.4] -0.1 [-0.7 to 3.1] -0.3 [-1.3 to 3.4] 28.9 [23.5 to 29.4] 31.1 [21.1 to 31.7] 225.0 [222.7 to 231.1] 3
sctp asn1 uring-busy 56.4 [56.0 to 58.4] 78.2 [75.1 to 78.3] -4.0 [-4.1 to -3.9] -3.5 [-3.7 to -3.0] 23.7 [23.4 to 25.3] 25.9 [22.9 to 26.5] 268.3 [266.3 to 276.8] 3
sctp asn1 uring-sqpoll-busy 1044.5 [995.9 to 1044.7] 1929.8 [1745.7 to 3017.1] 4.9 [4.9 to 5.0] 5.3 [5.2 to 5.3] 23.5 [23.0 to 23.6] 22.7 [22.3 to 22.9] 4325.7 [4322.6 to 4445.4] 3

Fastest link per transport

  • ipc/asn1: fastest is uring-busy (29.3 us), then uring (47.0 us, 38% slower). Against posix (48.3 us): 39% less.
  • tcp/asn1: fastest is uring-busy (39.8 us), then posix (45.0 us, 12% slower). Against posix (45.0 us): 12% less.
  • sctp/asn1: fastest is uring-busy (56.4 us), then posix (69.1 us, 18% slower). Against posix (69.1 us): 18% less.

These numbers are measured inside a GitHub Actions container and should be treated as an upper bound on E3AP's and the library's own latency, not a representative deployment measurement.

MPMC queue benchmark

MpmcQueue End-to-End Latency (SPSC, 30000 items per queue size)

Queue Capacity P50 (ns) P95 (ns) P99 (ns) P99.9 (ns)
16 562 593 614 13314
64 2456 3812 5847 27485
256 7592 12109 16889 19217
1024 5782 17407 25581 31174
4096 7816 15998 18086 19446

LockFreeQueue End-to-End Latency (SPSC, 30000 items per queue size)

Queue Capacity P50 (ns) P95 (ns) P99 (ns)
16 383 631 765
64 381 611 720
256 547 16707 19870
1024 518 7892 34899
4096 512 770 6104

Throughput (400000 items per configuration)

Configuration Queue Cap Throughput (Mops/s)
SPSC (1P×1C) 256 9.29
MPSC (4P×1C) 1024 5.87
MPMC (4P×4C) 4096 6.23
MPMC (8P×4C) 4096 4.29

Stress / Correctness (100000 items each)

Test Items Result
SPSC (1P×1C) 100000 ✅ PASS
MPSC (4P×1C) 100000 ✅ PASS
MPMC (4P×4C) 100000 ✅ PASS
MPMC (8P×8C) 100000 ✅ PASS

ubuntu-latest, Release build.

Docs

Doxygen warnings

0 warning(s) in the files this pull request changes, which fail the check; 48 more elsewhere, which do not. The full list is in the job log and the doxygen-warnings artifact of the Docs workflow.

Gaps in files this pull request does not change, by file:

Warnings File
17 include/libe3/logger.hpp
9 include/libe3/dapp_subscription_state.hpp
7 include/libe3/lockfree_queue.hpp
6 include/libe3/mpmc_queue.hpp
1 include/libe3/error_codes.h
1 include/libe3/libe3.hpp
1 src/connector/zmq_connector.hpp

Ready to merge (fast-forward only)

A maintainer can land the reviewed commits with:

git fetch origin
git checkout main && git merge --ff-only d429af715262fcf0a725079e175bcbf755d1cfad && git push origin main

Head: d429af715262fcf0a725079e175bcbf755d1cfad (branch feat/140-uring-connector). If --ff-only fails as non-fast-forward, the branch must be rebased on the latest main.

One comment per PR, rewritten in place once every workflow for d429af7 finished.

Red Hat Enterprise Linux 9 and OpenShift's nodes ship kernel.io_uring_disabled=2, which
refuses the io_uring link even to root. docs/connectors/uring.md gains a section with the
least that has to be opened: set kernel.io_uring_group to a group of its own, set
io_uring_disabled to 1, run the RAN or the dApp as a member of that group with no
capability, and put both values back afterwards (from a trap, so that a failed run does
not leave the node open), with the check that it took and that another user is still
refused, and what changes in a pod (a privileged container is needed to write the
sysctl, which is the host's and not namespaced, so it cannot be set from
securityContext.sysctls). The README gets the same commands in a short section of its
own, with a pointer to the page.

This is what was done on the cluster for the pod measurements: the window was open for
four minutes, the node read 2 and -1 again afterwards, a user outside the group was still
refused while it was open, and one in the group created rings with no capability.
Persisting the setting across reboots is described but was not tried.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Andrea Lacava <a.lacava@northeastern.edu>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant